[Feature][C++] Support mmap as an optional local file read backend - #920

Summary

• Add AUTO, MMAP, and PREAD local file read backends for the C++ reader.
• Implement POSIX mmap and Windows file mapping.
• Support automatic fallback and explicit mapping errors.
• Expose the read backend configuration through the C and Python APIs.
• Add correctness and lifecycle tests, documentation, and a read-backend benchmark.

Backend behavior

• AUTO prefers memory mapping for supported regular files and falls back to the positioned-read backend when mapping is unavailable.
• MMAP requires memory mapping and returns an explicit error if the file cannot be mapped.
• PREAD preserves the existing positioned-read behavior.
• The configured backend is captured when a reader opens a file, so changing the global configuration does not affect existing readers.
• A file must not be modified or truncated while it is open through the MMAP backend.

Benchmark

These are preliminary Windows Release results. They are not used as a performance threshold because relative performance depends on the access pattern, cache state, filesystem, storage device, and hardware.

Environment and protocol

• Platform: Windows 11
• Compiler: MinGW-w64 GCC 15.2
• Build: Release, -O3
• CPU: 16 logical processors
• Memory: 31.73 GiB
• Dataset: 8 TsFiles totaling 67.27 MiB
• Low-level access: single-threaded 64 KiB sequential ReadFile::read() scans
• Cache-state benchmark: 12 balanced rounds per state, alternating PREAD/MMAP execution order
• DataFrame benchmark: 6 rounds, 4 sequential epochs, and 32 random batches per round

Cache-state study

The low-level benchmark separates three different cache and mapping states:

Note: PageFaultCount is a process metric, not a disk I/O metric. PREAD copies file data through the kernel file cache into a user buffer, so file-page faults are generally not counted for the calling process. MMAP accesses mapped pages directly, so first-touch faults are counted for the process. Therefore, a PREAD value of zero does not mean that no disk I/O occurred.

The last two rows differ only in whether the same MMAP view was completely traversed before timing.

With a newly created mapping, the first sequential traversal incurred approximately 17,233 page faults and reached only 0.43x the throughput of PREAD. After pre-touching the same mapping, these page faults disappeared and MMAP throughput increased from approximately 2.54 GiB/s to 17.03 GiB/s, reaching 2.99x the PREAD throughput.

This explains why MMAP may be slower for the first complete sequential scan even when the OS file cache is already warm: the process still needs to establish page-table mappings when it touches the MMAP pages for the first time.

The approximate cold-cache state used a unique CopyFileExW(..., COPY_FILE_NO_BUFFERING) copy for each round instead of clearing the machine-wide standby list. Copy preparation time was excluded from the measured throughput. Storage-controller caches, filesystem metadata caches, and background system activity may still affect this result.

Windows PageFaultCount is a process-wide total and does not directly distinguish soft and hard page faults. In the warm-file-cache experiment, the MMAP page faults are inferred to be primarily soft faults because every file was completely scanned before opening the measured backend.

The current MMAP implementation still copies mapped bytes into the caller-provided buffer in ReadFile::read(), so this is not a zero-copy benchmark.

TsFileDataFrame end-to-end benchmark

The end-to-end benchmark represents a training-style workload:

• 8 TsFiles
• 1 device per file
• 8 DOUBLE measurements per file
• 131,072 timestamp rows per file
• 64 output columns in total
• 8,192 rows × 64 columns per batch
• 16 batches per sequential epoch
• 6 rounds per backend configuration
• 4 sequential epochs per round
• 32 random batches per round

For this training-style workload, MMAP improved end-to-end throughput by:

• approximately 27–30% with one query worker;
• approximately 11–12% with four query workers.

The relative improvement becomes smaller with four workers because decompression, decoding, NumPy materialization, scheduling, and parallel-query overhead account for a larger part of the total runtime.

Unlike the low-level sequential scan, the DataFrame reader performs many fine-grained metadata, chunk, and page reads. MMAP avoids repeated positioned-read calls in this access pattern, so it still improves the first DataFrame epoch despite the initial page-touch overhead.

Native stack-sampling observations

Periodic GDB native stack sampling was collected for both backends:

• Low-level PREAD scans contained ReadFile/NtReadFile in 94–99% of captured stacks.
• The corresponding MMAP scans contained no ReadFile/NtReadFile stacks and were dominated by the memory-copy path.
• In the DataFrame workload, ReadFile/NtReadFile appeared in 8.5–9.9% of PREAD stacks and in none of the MMAP stacks.
• Value-page decoding appeared in approximately 23–32% of DataFrame stacks for both backends.

These results confirm that the MMAP backend removes the repeated positioned-read path, while decompression and decoding remain substantial end-to-end costs.

The profiles were collected through periodic GDB interruption rather than ETW hardware sampling. Stack percentages describe the presence of a function in captured stacks and should not be interpreted as direct elapsed-time percentages.

Interpretation

MMAP is not faster for every workload.

A newly created mapping can be slower for a one-time sequential scan because the first traversal must establish the process page-table mappings. MMAP is more beneficial when:

• the same mapping is reused across multiple epochs;
• the workload performs many small or random reads;
• metadata, chunk, and page ranges are accessed repeatedly;
• a training or analytical workload repeatedly queries the same TsFiles.

PREAD can remain preferable for one-time sequential scans. Users can explicitly select PREAD for that access pattern.

AUTO is a compatibility-oriented fallback policy rather than a workload-adaptive performance policy: it prefers MMAP for supported regular files and falls back to PREAD only when mapping is unavailable.

Validation

• Added C++ tests for positioned reads, bounded MMAP reads, backend selection, resource release, repeated close, automatic fallback, required-MMAP errors, unsupported mapping, empty files, and invalid configuration.
• Added C API configuration round-trip and validation tests.
• Added Python API configuration and query-equivalence tests.
• C++ Spotless formatting completed successfully.
• The cache-state benchmark completed 72 formal low-level trials.
• The TsFileDataFrame benchmark completed 24 formal end-to-end trials.
• Eight PREAD/MMAP native stack profiles were collected and validated.

Closes #903